Property-Based Testing and Verification: A Catalog of Classroom Examples

For the past a decade, John Hughes of Chalmers University and Quviq has pioneered property-based testing in debugging industrial software. Three undergraduate courses at the University of Oklahoma have paralleled this work, but on the pedagogical side and with the additional goal of verification of properties by mathematical proof, using both paper-and-pencil methods and mechanized logic. An essential part of both efforts, and probably the most important part, is the formalization in predicate logic of the expectations of software developers. This is central to engineering software quality in the defect dimension. This report discusses property-based testing in course work and provides some examples from the classroom. Hughes has observed that software properties useful for testing functional software often (1) compare the results of different ways of computing the same thing or (2) check that forward and inverse transformations restore the original data. What percentage of useful software properties fall into these categories? Our collection of examples from course records may help shed some light on how closely pedagogy on property-based testing matches observations of relative frequencies of various categories of properties in industry and may also serve as an educational aid in learning to use property-based testing effectively in software development. 1 Property-Based Testing in the Classroom Students in the computer science program at the University of Oklahoma are required to take three courses in which expected software behaviors are expressed as formulas in predicate logic. In one of the courses (applied logic), the focus is on paper-and-pencil proofs that the properties hold. In the others (a twocourse sequence in software engineering), the focus is on software development, including testing and verification. For the past three years, all three courses have included project experience with automated testing on random data generated by the DoubleCheck tool in the Dracula programming environment [1], a facility similar to QuickCheck [2], but for software expressed in ACL2 [3], which includes a programming language based on Common Lisp. Used in its native form, the ACL2 mechanized logic requires properties to be expressed as theorems, and programmers can sometimes succeed in getting ? Partially supported by National Science Foundation Award Number 1016532.